DOCtTHENT RESUME 



ED 242 751 TH 840 162 



AUTHOR Caulley, Darrel N. 

TITLE Concept Analysis in Evaluation. Paper and Report 

Series No,_51, 

INSTITUTION Northwest Regional Educational Lab. ^ Pbrtlahd^ OR. 

Research on Evaluation Prbgrain^ 
SPONS AGENC7 National Inst, of Education (ED), Washington, DC. 
PUB DATE Jul 81 _ 

GRANT 400-80-0105 
NOTE 28p. 

PUB TYPE Reports - Descriptive (141) 



EDRS PRICE MFdl/PCd2 Plus Postage. 

DESCRIPTORS *Coinpensatory Education ; *Evaluation ; Evaluation 

Ffethods; *Evaluative Thinking; *Prograni Evaluation; 
Research Methodology 

IDENTIFIERS ^Concept Mapping; Evaluation Problems 



ABSTRACT 
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PREFftCE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluation methbdplogies for 
use in education. This document _is one of a series of papers and 
reports produced by program staff, visiting scholars, adjunct 
scholars/ and project collaborators— ^all members of a cooperative 
network of colleagues working on the development of new 
methodologies. 

What are the four different types of questions of concern in an 
evaluation study and how can concept questions be fecdgriized arid 
answered? These issues are addressed in this _paper which includes 
a description of three types of analysis (dif f ereritiatiori, 
generic, and conditions) of use in dealirig with concept questidris 
in evaluation. 

Nick li. Smith, Editor 
Paper arid Report Series 



4 



CONTENTS 



Page 

The Reiati^ of Concept Questions to Fact^ 

Technical f and Value Questions . . i i i . 1 

Strategies of Concept Analysis 5 

A Differentiation-Type Analysis 5 

A Generic-Type Analysis 8 

A Cbhditibhs-Tvrpe Analysis ............... 14 

Hw to Choose Which Strategy to Apply. ......... 17 

Concept Maps 18 

When Should Concept Analysis be Used? ......... 25 

Further Reading. 21 

Reference Notes. ........... i i ^ ••• 22 

References ... i 23 



CONCEPT ftNfttYSlS IN EVALUATION 



This paper is about thinking in relation to evaluation a 

how to bring clarity to our thoughts- The techpxques of corcept 

analysis were established over forty years ago, and though they 

have suffered from being tied too tightly to the apron strings of 

certain schools of modern philosophy, a great deal of headway has 

been made since then. In a quiet way> they have caused something 

like a revolution in the approach to questions of a certain type, 

I 

namely concept questions • 

1 

Concept analysis can provide the evaluator with methods which 
the evaiuator can leam to use in answering many of the more 
in%>brtaht and interesting questions which can be asked in 
relation to an evaluation. The importance of concept analysis to 
evaluation has been little realized. General and abstract 
questions cannot be tackled without the techniques of concept 
analysis in any but the most feeble and confused manner. It is 
little realized that the techniques can be learned quite easily. 

The paper begins by discussing the identification of concept 
questions and how they are different from fact, technical > and 
value questions. Following this, three strategies for concept 
analysis are discussed in some detciil. The paper ends with some 
discussion of the use of concept maps in evaluation i 

The P^ta-hton of Concept Ouegtions to 
Fact> Technical, and Value Questions 

What is a conc^t question and how is it different from other 
questions in an evaluation? There are at least four major types 
of questions asked in an evaluation: technical questions, fact 
questions, value cpiestions and concept questions. Ah evaluation 
is designed to answer value questions, and the other three types 



of questions have to be answered in order to answer the value 

I' 

.Jliestidnsi If a compensatory education program is to be 
evaluated, the following are examples of the four types of 
questions : 

Technical question ; How might the jef feet of this 
compensatory educatidh program be determined? 

_ ' _ 

Fact question ; Does the cbmpensatdry education prograffl 
have ah effect? 

Value question ; Is the cbihpehsatbry education program 
any good? 

CdnGe pt question ; Is compensatory f^^^^ion equivalent 
to the promotion of equality of educational opportunity? 

Technical questions are asked and answered on the way to 
detenninihg the answers to fact qiiestidhs; Technical questions 
are asked in relation to the design of an evaluation, and to the 
data gathering techniques usedi Typically, technical questions 
are "how" questions. 

In contrast to the other questions, only knowledge about the 
world is heeded to answer fact questidiisi The answers to value 
questions and concept questions, and possibly to technical 
questions, are matters of human chdice which is not true for fact 
questions. Value questions cannot be answered without reference 
to the answers to certain fact quest: ions i 

Value questions are questions regarding the worth of things. 
They can readily be recdgnized, since they contain evaluative or 
rating terms like "good," "efficient," "better^" "right" and 
"desirable." V&lue questidhs sdmetimes begin with the terms 
"ought" or "should" which are also evaluative terms. Such 
questidhs are prescriptive questions, i^ei questions asking vihat 
to do. If seme action ought td be taken, then it is either right 
or desirable to take the action. A value question is a questidh 
abdtat sdme value object, e.g., cdmpensatdry education programs. 
While the answers to value questions are not dependent upon nor 
derivable ffdm facts aldne, the answers to value questions differ 
when there are differences in facts and these facts must be 
relevant to the answer. 



Concept questions are questions about i n i aaning * In the above 
example of a concept question> in order to answer the question 
it is necessary to explicate the meaning of the concepts of 
"compensatory education" and "equality of educational 
opportunity. " Explication of the meaning of concepts is Ichbwh as 
concept analysis. Concept analysis is not^ simply giving 
definitions to the meanings of terms or looking up their meahirigs 
in dictionaries. Some philosophers claim that the analysis of a 
concept is the description of its use. Complete descriptions 
specify the context or contexts of usage, different meanings in 
different contexts^ different thoughts which are possible with 
different uses, and exhibits of different applications of the 
concept iGbwin/ 1979). 

How are the four types of questions related? The aim of 
evaluation is to answer questions of value, arid to do this one 
needs the answers to questions of fact. Gathering facts requires 
technique and, thus, the answers to technical questions. In the 
:^bove examples^ the technical^ fact and value questions included 
the concept cf "compjehsatdry educations" Unless one is clear 
about this concept > one cannot answer the technical, fact and 
value questions. Thus it is not only important to isolate the 
concept questions but to deal with them first * Concept questions 
are the most basic and fundamental of all the questions. 

It is only rarely that one is presented with a question of 
concept in pure form such as in the exaii4)le giveii above. 
Consider the following two questions. Does conipensatdry 
education result in equality of educational opportunity? Is 
compensatory education inpori:ant for equality of educational 
opportunity? The first question is a fact question and the 
second question^ a value cfuastioh. In order to answer either of 
these questions it is necessary first to carry out a conceptual 
analysis to explicate the meaning of "cdmpensatdry education" and 
"equality of educational opportunity." Thus, in order to answer 
technical, fact or value questions , it is recommended that 
conceptual analysis be carried out first. Doing prior conceptual 
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analysis will shar.peri subsequent inquiry i Without it we just 
proceed at om current level of liriderstaridirig. 

Consider the example: Suppose an evaluator said, "That is a 
good program." We would probably want to ask, "What do you mean 
by a good program?" This is a question of concept because we 
want to know what counts as a "good progaih* " Yet it would be 
wrong to say that we are asking for the meaning of the word 
"good." "Good" is a very cdininbh word, which we use correctly 
every day. It means, roughly > "to be commended" or "to be 
approved" or "desirable." We know this already. Yet we still 
ask, "What do you mean by a good program?" Questions of concept 
are lidt concerned with t^ie^ meaning of a wordi Words do not have 
only one meaning. Indeed, in a sense they do not have a meaning 
in their own right at ail, but only in so fax as people use .them 
in different ways. It is better to say that we are dbncemed 
with actual and possible uses of words. That is why it is no use 
Ibbk^hg up the word in a olctidnary . It will not help when we 
ask, "what do you mesan, a good program?" What we are really 
saying is, "What counts as a good program with you?" br> "What 
are the criteria fbr a good program?" (Wilson, 196j ) 

Answering a concept question is not a matter of defining 
terms. Fbr the whole point of asking such questiois is that the 
definition of these words is unclear. Or> we might rather say 
that they db hot have defiiiitibhs, but only usesi 

Concept questions are not technical > fact or value 
questions. Nor are they questions cdn'^em^d witn :the- meaning of 
words > or the definitions of words. They are concerned with the 
useis of words and with the criteria or principles by which those 
uses are determined. 

The analysis of a concept is the descr iptibr of :ts use . 
It is_ describing when the concept applies, when it does 
not, how its sxibtle nuances incline us tb think one way 
or ahbther when we use it, the delicate differences of 
meaning it receives in different cbhtexts, and how the 
likenesses and differences between those contexts lead us 
to one or another use of the concept (Green, 1971 )i 
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How does concept analysis fit into the overall evaluation 
design? since the answers to value ^ fact, and technical 
questions depend on the answers to concept (questions / concept 
analysis should be done very early in the evaluation • Concept 
analysis should be done when the evaluation is being planned. 
Who should do the concept analysis? The evaiuator should be able 
to do a concept analysis carrying out the instructions given in 
the rest of this paper. If funds were available and the 
evaluation was a large ohe^ a philosopher could be employed to 
carry out the concept analysis. 

Strategies of Concept Analysis 

Soxtis (1978) identified three ba^c strategies for doing 
concept analysis: a differentiation-type analysis, a 
generic-type analysis^ and a conditions-tj^e analysis. The next 
section of .the paper will consist of a discussion of each of the 
three types. 

A biff erehtiatidn-Type Analysis 

Making distihctibhs is what this strategy is all about. The 
strategy proceeds by asking such questions as, "What are the 
different uses of the concept X?" or, "What are the various types 
of concept X?" and then seeking the distinguishing riiarks that 
separate these ideas. The intended result is a clearer idea of 
the logical terrain covered by different meanings of a concept. 
To ili^atrato differentiation-types analysis, the concept of 
"teaching" will b.3 analyzed.* 

Step^ Search for the dominant standard uses of the concept by 
means of examples. 
Teaching is a practical activity likes painting a room or 
baking a cake. In order to discern the dominant standard uses of 
the concept of "teaching>" we might list what a teacher does. 



♦The analysis described is that given by Green (1971). 
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9. 


Laughs 


2. 


Asks questions 


10 . 


Collects money 


3. 


Eva luates 


11. 


Motivates 


4. 


Takes attendance 


12. 


Consults with parent 


5 • 


Defines 


X J . 


ijxves reasuns 


6. 


Answers questions 


14 i 


Demonstrates 


7. 


Disciplines 


15; 


Drinks coffee 


8. 


Patrols the halls 






We 


clearly could make a 


vezy long 


list if we wanted to. 



the outset we would not want to include "Laughs" or "Drinks 
coffee, " since a person other than a teacher also carries out 
these activities. The fact that wc can readily make a list of 
what a teacher does means that we already know what teaching is. 
Green (1971) states: 

• • • the analysis of a concept cannot even begin unless 
we already have in mind some rough idea of what the 
concept is. . . . the aim is hot to invent some new 
concept or idea of teaching, or even to specify what 

people ought to mean by "teaching." The objective is 

rather to study, clarify, arid more thoroughly understand 
the idea of teachirig that we already have; (p. 3) 

Step II Intuitively classify or categorize the uses of the 
concept into types. 
Using an expanded version of the list> the activities of 
teachers can be classified into three types: the logical acts, 
the strategic acts and the institutional acts. The fbllbwirig is 
Green's {1971, p. 4) classification: 



The Logical ActB The Strategic Acj^ The institutional Acts 

JExplairiirig Motivating Collecting aoney 

Concliidirig Counseling Chaperoning 

Inferring Evaluating Patrolling the hall 

Giving reasons Planning Attending meetings 

Amassing evidence Encouraging Taking attendance 

Dembristfatirig Disciplining Keeping reports 



Step III Search for distinguishing marks of each type which can 
be used to clearly separate types. 
The logical acts refer primarly to the element of thinking or 
reasoning in the conduct bf teachings The strategic acts have to 
do with the teacher's plan or strategy and the ways material is 
organized bf students are directed in the course of teaching. 
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ihstitutidhal acts are those which are performed as a result of 
the way the teacher's work is organized by the school as an 
ihstitutidri. 

The institutional acts of teaching may go on independently of 
the activity of teaching. Teaching does not require the 
ihstitutidhal arrangements associated with schools. Teaching can 
occur between mother and daughter in the home. Quite typically, 
a mother teaches a daughter cooking and sewing. There is no need 
for the mother to collect money/ patrol the halls, attend 
meetings/ and so on. Teaching may go on even when institutional 
acts of teaching are hot going on. BUt this is not true for the 
logical and strategic acts of teaching. The absence df the 
strategic and logical acts df teaching would count strongly 
against the view that tesaching was going oh. 

The logical and strategic acts of teaching differ not only 
from the institutional acts, but also from each other. Learning 
car t£Oce place without the logical acts of teaching. For 
example/ a teacher can give a gddd explanation df scsne concept/ 
but at the same time the students may not learn or understand the 
concept. The e^H^lahatidh/ while it is a gC5od one/ might not be 
understandable by elementary school students but may be suitable 
for a graduate seminar. Thus it can be decided whether a logical 
act is good without considering whether anyone learns frdm it. 
The perfdnhahce df the Idgical acts of teaching can be decided on 
by an evaluator oh Idgical grounds. In contrast/ perfdrtnances of 
the strategic acts of teaching are evaluated by their 
consequences for learning. Logical acts and strategic acts are 
evaluated differently. Logical acts require a khdwledge df the 
methods of knowing. Strategic acts require a knowledge of human 
behavior and motivation. 

Step IV Test the typdldgy develdped by means df examples and 
counter examples. 
The typdldgy wll be tested by looking at five examples of 
teacher acts: questioning/ defining^ laughter/ consulting 
parents/ and cdmparihg. QUestidning can be classified as a 
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strategic act because it refers to a way of directing students 
during the course of teaching; Defining is a logical act because 
it refers to reasoning in the conduct of teaching. Laughter does 
not fall under any one of the types of teacher acts* Laughter is 
not ah act of reasoning or thinking (logical act), a way of 
directing students (strategic act), or a way of performing which 
is shaped by the school (institutional act). Laughter is just 
one of a number of teaching acts that cannot be classified by the 
typology. Other examples are sitting, walking, running; and 
yawning. Can the typology be changed to accommodate such 
examples? The answer is that it could, siinply by adding an 
"other" category. However, this "other" category is hot 
particularly useful in explicating the meaning of teaching. 
Consulting parents is an institutional act> since it is performed 
as a result of the weiy the teacher's work is organized by the 
school as ah institution. Cdn^arihg is a logical act, since it 
refers prijaariiy to the element of thinking or feasbhihg in the 
conduct of teaching. 

This completes the four steps of doing a dif f erehtatibh type 
cdhcept analysis. The aim of differentiation-type analysis is to 
clarify and thus make more useful a cdhcept by poihtihg to the 
different basic meanings it has. The form the prior question 
takes is "what are the basic sehses of X?" First, seek examples 
from ordinary language which will display different uses or 
meanings of the term. The next step is to look for boundries 
between different uses of the term em.d come up with a t^^dldgy. 
The hext step is to search for distinguishing marks of each type 
when it can be used to clearly separate types. Fihally, these 
distihguishihg marks are tested and refined by means of examples 
and counter examples until a useful set df basic uses or meanings 
or types result. 



Generic-type analysis is a means of clarifyihg a concept by 
identifying its key characteristics. The analysis tries to 
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identify the general or generic characteristics of a concept. 
The basic conceptraal question^ "What is ah X?" is asked and taken 
to mean, "What are the baisic features which make an X (species) 
ah X (genus) and provide the criteria for distinguishing Xs from 
fidn-Xs?" Dif f erentiation-t^^e analysis distinguishes cimong the 
basic meanings of a concept, whereas generic-type analysis points 
to the necessaZj features of a concept. 

The techniques of generic-type analysis will be illustrated 
by an example taken from an evaluation. An evaluator was doing 
ah evaluation of a program within an institution for teacher 
training. There was a difference of opinion among the staff over 
whether teaching was a step-by-step procedure that could be 
taught to student teachers or whether it was ah art that could be 
taught by eraulationi The basic disagreement was sximmed up by the 
question, "Is teaching a science?" In order to answer the 
question, the evaluator carried out a concept analysis of the 
qtiestidh. 

Step Z - Isolate the conceptual question or questions from the 
rest of the question. 
It will be noticed that the question is a mixed question, 
involving both the knowledge of the nature of teaching and an 
understanding of the concept of science. We will deal with the 
question of cohc^t first. 

St.ep_^X Apply the following techniques to the concept. 

A. Model cases . Put forth ah exemplary example of the 
concepts-— that is^ ah example which we are absolutely sure is an 
instance of the concept, something of which we could say, "Well, 
if that isn't ah example of sd-and-sd, then nothing is." 

An exemplary model of a science is "the queen of the 
sciences," physics. Physics has a logical structure going from 
basic definitions to the fdrmulation of general laws. 

B* Contrarj^ cagea . Put forth an example that is a contrary 
case> one that certainly is hot ah instanced 

A contrary excmiple of science would be where sdmebhe painted 
a picture of a physicist at work. This activity is not like the 
sciences but belongs to the arts. 

9 
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C> Related casr s » Piit forth ah ^'^aii^le that is a related or 
similar case; one that is in some way iii5)ortantly connected. 

A related example to science is knowledge ^ though all 
knowledge is hot science. A more closely related example is 
knowledge of nature i However, such people as Wordsworth or 
Cbhstable or eveh fanners could be said to know nature. They 
have factual knowledge # but they are not able to frame laws and 
hypotheses^ and they do hot do experiments i 

Dm Borderline cases . Put forth a case where we are hot sure 
that it is ah example of the concept. The point of such cases is 
to elucidate the nature of the concept by cbhtinually facihg 
ourselves with different cases which lie on the borderline of the 
concept; what we might call odd or queer cases. By seeihg what 
makes them odd or queer* we ccnne to sev why the true cases are 
not odd or queer^ and hence what makes them true cases — what the 
central criteria of the concept really are. 

A borderline case of science would be psychology. 
Psychologists do freune laws and hypotheses and they do 
experiments. However* it is still hot certain that psychology is 
a science* since it can be argued that psychology does hot tell 
us anything that we don't already knowi Another borderline 
example is that of meteorology* or the predicting of the 
weather i Is it a science? It seems to depend on whether 
meteorology can predict the weather better than the ordinary 
human being. So prediction is an important criterion of sciehce* 
as are experiments* theories* and hypotheses i 

E. Invented cases . Wilson (1963) states: "Sometimes it is 
necessary to invent cases which are in practice qxrtte outside our 
ordinary experience* singly because our ordihary experience does 
not proT^de us with enough different instsmces to clarify the 
cbhcept. " (p. 32) 

Suppose I look into a crystal ball and predict accurately the 
winner of the Kehtucky Derby every year. Suppose I have ho idea 
how I do it and si:^pose that I conduct no es^eriments. I just 
look* and then tell you the wihher. Is this sciehce? No* not at 
all. Perhaps it would be if I added to the crystal ball some 
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equipment and did some experiments. Suppose I added a vast 
amount of dquipmeht to the cyrstal ball, including wires and 
tubes, would this make it science? No# since I had dressed the 
ball to look like science i I had not arrived at my predictions 
by reasoning and observation. The equipment were riot really 
connected with ray predictions. Some more criteria for science 
are discernible. Firstly, the activity has to tell us more than 
we already know. Secondly, it has to do this not by guesswork 
but by observation, experiment, arid the testing of hypotheses by 
experiment . 

F^- ^Social context; Concept questions are not asked in a 
vacuum but are asked under particular circumstarices . The nature 
of these circumstances is in^jortant for understanding the concept 
question. We heed to state or imagirie, who would be likely to 
make such a statement > w}^ he or she would want to make it, arid 
wheri he or she would most likely make it; 

In the case of "Is teaching a science?" we kriow the 
circumstarices surrounding the question. There was a difference 
of dpihidh among the staff of a teacher trairiirig institution over 
whether teaching was a step-by-step prdc^?dufe that could be 
taught to student teachers dr whether it was ari art that could be 
taught by emulation; 

G. Underlying anxiety . Closely connected with the 
importance of looking at the social context of a question or 
statemerits is the impbrtarice of looking at the mood or feelings 
of the person (s) who makes it. 

Iri relation to the question under consideration, there is an 
-andef lying anxiety that teaching might riot be a scierice. There 
is an implicit belief that if any activity is a science^ this 
then increases the prestige of the activity. Those who answer in 
the negative may be worried if teaching is taught as a scierice, 
teachirig may itself become less warm and spontaneous. 

H. Practical results . Since sdme conceptual questidris have 
rid right or wrong answer, we wonder if such questions have any 
meariirig or pdirit to them at all. One way to get at the point or 
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meaning is to ask what would be the practicai results in everyday 
life if we answered, "Yes," or "No" to the questioni 

If we answer "Yes" to the question "Is teaching a science?" 
then teaching would involve ah organized body of knowledge with 
facts and theories based on experiments. There would be 
professors of teaching^ teaching from textbooks on the science of 
teaching. If we answer, "No" to the question, then teacher 
trainees cannot be taught teaching by laying down a step-by-step 
procedure for carrying out teaching. 

I ; Rgsults xn language . Since words are not used without 
ambiguity, and it is hot always possible to say what the^ meaning 
of a word is, we m^ often be left with the situation in which we 
have to say, "Well, if ydii meah abc by sd-ahd-sd, then the answer 
is thisi But if you mean xyz > then the answer is that." Wilson 
(1963) states: 



... we have to look at the "results in language" when 
chbosihg meanings for words or delineating ^^'^^^^or 
concepts: we have td pick the most useful criteria for 
the concepts Thus, when (but only when) we have analyzed 
the concept and noted the whole wealth of possible 
instances df it, we may often have to say at the end, 
"itoid ail these possible meanings of the word so-and-so, 
it seems most sensible and useful to make it mean 
such-and-such: for in this way we_shall be able td use 
the word to its fullest advantage." (p. 37) 

If we find that teaching satisfies some criteria of science 

but not others, we might want td call it a science, even if this 

means stretching the concept of science beyond its normal 

limits. Oh the other hand, if teaching satisfies none of the 

criteria, or only the less important bhes> we have no reason to 

extend the concept of science td include iti 

Step tXl. Conduct a dialogue with yourself about the concept. 

Ask yourself questions and answer them. Invent new cases 
when that seems helpful. Go back to the application of the 
techniques in the last step. The following is an interior 
dialogue over the question, "Is teaching a science?" 

What are the criteria for science? Firstly, a science 
must be able to make predictions beyond that made by the 
average man. Anyone can predict a storm from looking at 
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the clcSuds. But for meteorology to he a science it must 
be able to predict rain wheii the average man cannot i 
Since ah average man may be able to guess correctly at 
times, meteorology has to predict successfully in a 
consistent manner • 

Secondly, the prediction must be made the basis of 
iome organized technique, observation/^ reasoning or 
experiment. On this basis one would rule out crystal 
ball gazing. 

Thirdly, an ability to ea^iain is a hec 2ssary criterion 
of science. Toulmtn (1961) states: 

Fbrecastihg> then, is a craft or technology, an 
application of science rather than the kernel of 
science itself* If a technique of fbrecastihg is 
successful/ that i3 one more fact/ which scientists 
must try to ej^laih, and may succeed in explaining i 
Yet a hovel and successful theory may lead to no 

increase in biir forecasting skill; while, 

altertiatively , a successful f orecasting^technique 
may remain for centuries without any scientific 
basis. (p« 36) 



Take an example from early astronomy where there was a 
high level of ability to predict but notto es^lain. 
Early astronomy observed the stars and planets and 
noticed that they move in certain regular orbits over 
certain time periods. By dint of constant bbservatidn, 
but no theorizing about causes , early astronomy was able 
to predict accurately what planets will be in what part 
of the sky at certain times. While science is expected 
to be able to predict, it is hot ah essential criterion 
but the ability to ea^laih is. 

Is teaching a science? Teaching is a practical activity, 
not an organized body of knowledge. It may draw on 
concepts from psycho logy and sociology which are 
organized bodies of knowledge/ but there are no 
step— by-step teaching procedures which are based on 
organized knowledge drawn from observation/ experiment> 
or theories. 

Is teaching capable of making predictions? Teaching is a 
practical activity which does not aim to make 
predictions, ft necessary criterion of a science is that 
it has the ability to explain phencaneha» Teaching does 
not aim to make explanations. 



The question, "Is teaching a science?" might be taken to 

mean/ "Is teaching scientific?" That is, is teaching 

based on scientific ideas and theories? Some ideas about 
teaching are drawn from psychology and sociology, but the 
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connections are rather loose. There is some doubt 
whether these are Jciahces/ since they piit forward 
knowledge which ia already possessed by the average man. 

Step IV List oh paper the points you are going to make and the 
conclusion you are going to reach. 
Ai Science is more than the knowledge possessed by the 
average man. 

Bi Science is an organized body of factual knowledge and 
theory about the phenomena of nature and is logically unlike art 
or aesthetic appreciation. 

C. The criteria for science se>5m to be. 

1. The ability to predict with reasonable consistency 
in areas where the ordinary man cannot do soi 

2. Tne predictions must be firmly based on observation, 
theories and experiments, in such a way that they can be seen to 
issue from these. 

3i The ability to escplaxn phenomena i 

D. Teaching is hot a science, since hone of the above 
criteria are satisfied. 

E. The question, "Is teaching a science?" might be phrased 
as "Is teaching based on scieiitific ideas and theories?" 
However, teaching is b:Uy loosely tied to scientific ideas and 
theories drawn from psychology and sociology which themselves may 
be considered on the borderline between science and non-science. 

A Conditions-Type Analysis 

A generic-type analysis or a differentiation-type analysis 
may hot always seem useful or possible i For example, model cases 
from which to draw potential generic features do hot seem to be 
readily available i We may not be sure what would count as a 
model case of "teachihg>" "eaqplainihg, " or "understanding." The 
point of departure for a conditions-type analysis is to puzzle 
over the context appropriate for the use of the concept and hot 
so niuch over its more acontextuai generic meaning or its 
different senses {Soltis, 1978). 
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The fdria the conceptual question Ec3f a conditions-type 
analysis] takes is: "Under what conditions or under what 
circumstances would it be true to say that X?" (eigi, to 
say that someone knows scanet hi ng . . . . . ) The point of 
the general strate^^ of a conditions-type analysis^ 
try to identify the necessary and stifficient conditions 
required to prc^erly apply term X and to test by example 
cases in which a cohditibh holds but application of the 
concept is withheld. This helps locate the need for 
additional conditions (Sbltis, 1978, p. 51). 

A cbmmbh problem in evaluation is to know what should count 
as a case of sxiccessful learning and teaching of some khbwledge 
that-. Knowing 4:hat^ can be contrasted with knowing how . The 
distinction may be stated as a logical difference between khbwihg 
facts or having information (that), and possessing skills , or 
being able to perform certain operations {how) . The f ollbwihg 
are the steps of a conditions-type analysis applied to the 
conceptual question: "Under what cbhditibhs or Under what 
circumstances would it be true to say that someone knows 
sbmethihg?" 

Step I Identify a good candidate for being a hecesisary 
condition of X happening c X being present in a 
situation. 

ftt least one condition that must hold for the ascription of 
khbwledge to sbmebhe is that the proposition must be true. 
Soltis (1978) states: "Truth is a fundamental condition bf 
knowledge. Although I might be taught and, in fact, learn that 
the earth is flat or that water boils at 0°C, such prbpbsitibhs 
obviously db hot constitute knowledge about the earth or about 
the properties of water. "Tr\ae khbwledge" is a redundancy, for 
we only count as knowledge that which we also count as tme." 
(p. 49) 

gtgp tX By altering the context> try to find ah exan^le where 
the condition hblds, but X or Xing is not present. 
One could alter the context to a case where a student made a 
claim tb know on the basis of making a lucky guess and being 
right. The cohditibh identified in Step I (i-ei the proposition 
must be true) stili holds but knowing is not present. Thus we 
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have foiirid ah example where the cbhditibh holds, but knowing is 
not present. 

Step III Revise or modify the cbhditibri to meet the context 

problem or tease out from the altered context another 
cbhditibh ahd test it« 
We tease out a new condition that we need evidence of what 
the studeht claijns tb khbw. Ih essence/ this evidence condition 
rec[uires that the student back up his or her beliefs and provide 
sdue grounds or warrant for what he or she takes to be true. It 
requires that the student provide reasons/ evidence/ or proof for 
what the student asserts he or she taows. Pedagogically 
speaking/ we generally expect that students shbuld learn what 
constitutes adequate evidence > good reasons > or reliable proofs 
fbr sbme beliefs/ so that they may be better prepared to acquire 
knowledge oh their own and/ in the future/ to test their bwh 
knowledge claim and those of others (Soitis/ 1978). 
Step IV Test the necessity and sufficiency of the cbhditiohs 
arrived ati 

But what constitutes adequate evidence br prbbf ? It is hot 
enough to be in possession of the evidence. One must understand 
the prbbf ahd see its pbiht. An example of a mystery story 
e^^iains this. Prior to the super sleuth's inevitable 
ahhbuhcemeht bf the murderer's idehtity/ all the clues / all the 
evidence has been woven into the details of the story. The 
reader has the evidence/ but it takes the detective to pull it 
all together ih a persuasive patterh befbre the reader sees that 
only the butter couid have done it (Soltis/ 1978). Thus the 
cohditioh that we heed evidehce bf what the student claims to 
know is not a sufficient condition. 

This completes the discussion of the three types of concept 
analysis: diff erehtiatioh-ti»pe/ geheric-type/ ahd 
conditions-type i Step hy step procedures were given for each 
type. It might be found ih scihe analyses that it is advantageous 
to use two or more of the types in combination. 
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How to Choose Which Strategy to Apply 

Choosing which of the three strategies to apply depends on 
the nature of the conceptual question being asked. Most of the 
cbhc ipt questions asked in an evaluation will reqpiire a 
generic-type strategy. They will be of the form, "What is X7" or 
"What is meant by X7" e.g., "What is compensatory education?" or 
"What isi meant by equality of educational opportunity?" In other 
words: "What are the basic features which make cou.^ ^n-;atory 
education a form of education and provide the criteria for 
distinguishing coarpensatory education from nbh-compehsatory 
education?" Generic-type analysis is a means of clarifying a 
concept by identifying its key characteristics. 

if the concept question in ah evaluation requires making 
distinctions, then a differentiation-type analysis is called 
for. For example, the question, "What is the difference between 
teaching and indbctri nation? " requires one to maDce distinctions 
between different forms of education- In differentiation-type 
analysis we clarify and thus make more useful a concept by 
pointing to the different basic mi»anings it has. The concept 
question takes the form "What are the basic senses of X7" The 
strategy is to seek exaimples from ordinary language which will 
display different uses or meanings of the term. 

A generic-type analysis or a differentiation- type analysis 
may not always seem useful or possible so that a cohditibris-type 
analysis may be carried out. For example, model cases from which 
to draw potential generic features do hot seem to be '-aadily 
available. Ah example of a conditions-type analysis question 
is: "Under what conditions would it be true to say that someone 
knows something?" The purpose of a conditions-type analysis^ 
theh> is tb produce the set of necessary and sufficient 
conditions for the proper application of a concept to any of its 
many and varied instances. A generic-type analysis, on the other 
hand, seeks to determine the essential characteristics of the 
paradigmatic form of a concept (Soltis, 1978). 
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Concept Maps 

Concept maps portray the conceptual structure of an - 
evaluation study. Concept, maps alaborate and make clear how the 
concepts in an evaluation study are related to one another^ A 
concept map may be drawn after an evaluation is complete for the 
purpose of carrying out a meta-evaluation of the evaluation 
study. Alternatively a concept map may be drawn during the 
planning stage of an evaluation ±ii order to clarify the 
conceptual structure of the evaluation. In order to understand 
what a concept map ±3, an example of one will be described. 

Gbwih (1979) has drawn a concept map for the evaluation study 

conducted by Kaplan (1966) and titled "Head Start Eb^erience and 

the Development of Skills and Abilities in kindergarten 

Children'* • To understand the concept map it will be necessary to 

give a brief description of 3Kaplan's study. The following 

description was written by Kaplan; 

The purpose of thie study was to assess whether the 
children who particxpated in Project Head Start were 
better prepared for kindergarten than those who did hot 
participate in regard to verbal cbinmiihicatibh* visual 
discrimination and naxning^ and motor cobrdihation 
skills. It was hypothesized that kindergarten children 
who participated in Project. Head Start would be st^erior 
in verbal cbmnhinicatibn abilities as measured by verbal 
fluency/ verbal usage, entihciatioh, ability to structure 
sentences/ and ability to tell a story; that they wbuld 
be sx^eribjr in the visual discriiaihatioh and naming of 
colors and shapes; and that they would be superibr in 
motor coordination skill/ as measured drawing figures, 
coloring/ cutting/ and buttoning their clothing. 

The sxibjects were seventy kindergarten children between 
the ages of four and five who were identified on the 
basis of whether or hot they had participated in the Head 
Start program during the summer of 196S; Thirtyfive 
children who had participated in this project were paired 
with thirty«*five children who had not participated in the 
project using sex, age/ etbiiic t^ackground/ language 
spoken in the home/ age of siblings/ and preschool 
sxperience as the criteria. 

The subjects were then compared with respect tb their 
vexbal coraoaunication abilities. They were asked to tel^ 
everything they saw in a sample picture and tb tell a 
story about that picture. They were rated according to 




the ability to structure sentences ^ story-; telling 

ability, verbal fluency, verbal asage, and enunciation; 

The s'objiects in each group were also compared in their 
visual discrimination abilities. and three 

shapes were presented to each chif:d who was then asked to 
4: ^ine each color and shape. The children's productions 
provided data to evciiuate their cutting, coloring, and 
drawing •skills. The children's ability to button their 
own clothing was also observed. 

The present findings support the current view that 
cuituraily deprived children benefit from pireschool 

enrichment programs. It was suggested that future 

research should further examine, the values of preschool 
compensatory programs and establish an appropriate 
curriculum. Longitudinal studies are needed in order to 
ascertain the long- term benefits of such a program. 
(Millman & Gbwin, 1974, j^). 37, 38) 
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The concept map shbw3 the ihtercbhhectibn of concepts as they 
were used in the evaiuatibh study. The irkDst general concepts are 
put at the top of the page and there is a transition to the most 
specific at the bottcm of the page. As one moves cfcwri the page, 
the key concepts move toward actual events. In addition there is 
a loss of abstraction, a gain in precision, and an increased 
likelihood the events of interest will be trivialized by the 
selection of indicators as they are operationally defined. 

Doing a concept map helps to reveal the thinking and 
reasoning behind a study. If the concept map is drawn during the 
planning stage of an evaluation, the evaiuator can detormine 
where his or her reasoning is weak or ineffective. It is 
important to do this since the whole conduct of ah evaluation is 
dependent on the reasoning behind the evaluation. Other examples 
of concept maps are given by Gowih and Green {Note 1) , and Lane 
(Note 2). 

When Should Concept Analysis be Used ? 

Constructing a concept map would be helpful in clarifying the 
reasoning and thinking which lurks behind an evaluation. Doing a 
concept map causes the evaiuator to exeimihe and to be more 
explicit about his or her thinking, ft concept map will often 
help the evaiuator discover flaws in his or her thinking and 
reasoning. Thus it is recommended that a concept map would be ah 
aid to doing any evaluation. 

When the questions in ah evaluatidh contain abstract terms 
then it is recommended that one of the three strategies of 
concept analysis be carried but. Cbhcept analysis is also needed 
when people talk past each other using the same terms or when 
different pebple interpret the same data differently. 

The generic-type analysis was illustrated by ah example of ah 
evaluation in f^ich the concept question asked was "is teaching a 
science?" This example suggests the difficulty of such ah 
analysis and one needs to decide a priori whether or not the 
analysis is really warranted. 



20 



Further Reading 

For further details on concept analysis^ thre books are 
rec^ended. Thomas (3 reen's (1971} book The Ac tivities of 
Teaching contains exajnpies of concept analysis. Chapter One 
examines how concept analysis is carried but. It was from 
Chapter One that the differentiation-type analysis given in this 
paper was taken, John Wilson's (1963) book Thinkings With 
Concepts contains many examples of generic-type analysis. This 
book was written for high school students and the exposition is 
very clear. Jonas Soltis' (1978) book Ah Introduction to the 




onal Concepts gives examples of each of the 



three strategies for concept analysis. 
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